{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_boston\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import mean_squared_error, r2_score\n",
    "from sklearn.preprocessing import PolynomialFeatures, StandardScaler\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".. _boston_dataset:\n",
      "\n",
      "Boston house prices dataset\n",
      "---------------------------\n",
      "\n",
      "**Data Set Characteristics:**  \n",
      "\n",
      "    :Number of Instances: 506 \n",
      "\n",
      "    :Number of Attributes: 13 numeric/categorical predictive. Median Value (attribute 14) is usually the target.\n",
      "\n",
      "    :Attribute Information (in order):\n",
      "        - CRIM     per capita crime rate by town\n",
      "        - ZN       proportion of residential land zoned for lots over 25,000 sq.ft.\n",
      "        - INDUS    proportion of non-retail business acres per town\n",
      "        - CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)\n",
      "        - NOX      nitric oxides concentration (parts per 10 million)\n",
      "        - RM       average number of rooms per dwelling\n",
      "        - AGE      proportion of owner-occupied units built prior to 1940\n",
      "        - DIS      weighted distances to five Boston employment centres\n",
      "        - RAD      index of accessibility to radial highways\n",
      "        - TAX      full-value property-tax rate per $10,000\n",
      "        - PTRATIO  pupil-teacher ratio by town\n",
      "        - B        1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town\n",
      "        - LSTAT    % lower status of the population\n",
      "        - MEDV     Median value of owner-occupied homes in $1000's\n",
      "\n",
      "    :Missing Attribute Values: None\n",
      "\n",
      "    :Creator: Harrison, D. and Rubinfeld, D.L.\n",
      "\n",
      "This is a copy of UCI ML housing dataset.\n",
      "https://archive.ics.uci.edu/ml/machine-learning-databases/housing/\n",
      "\n",
      "\n",
      "This dataset was taken from the StatLib library which is maintained at Carnegie Mellon University.\n",
      "\n",
      "The Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonic\n",
      "prices and the demand for clean air', J. Environ. Economics & Management,\n",
      "vol.5, 81-102, 1978.   Used in Belsley, Kuh & Welsch, 'Regression diagnostics\n",
      "...', Wiley, 1980.   N.B. Various transformations are used in the table on\n",
      "pages 244-261 of the latter.\n",
      "\n",
      "The Boston house-price data has been used in many machine learning papers that address regression\n",
      "problems.   \n",
      "     \n",
      ".. topic:: References\n",
      "\n",
      "   - Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261.\n",
      "   - Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "boston = load_boston()\n",
    "X, y = boston.data, boston.target\n",
    "df_boston = pd.DataFrame(boston.data, columns=boston.feature_names)\n",
    "df_boston['PRICE'] = y\n",
    "print(boston.DESCR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    :Attribute Information (in order):  \n",
    "        - CRIM     Уровень преступности на душу населения по городу  \n",
    "        - ZN       Доля жилой земли, зонированной для участков площадью более 25 000 кв. футов  \n",
    "        - INDUS    Доля акров для промышленности (заводы, предприятия)  \n",
    "        - CHAS     Фиктивная переменная (1, если тракт ограничивает реку; 0 в противном случае)  \n",
    "        - NOX      Концентрация оксидов азота NOX (частей на 10 млн.)  \n",
    "        - RM       Среднее количество комнат в одном жилом помещении  \n",
    "        - AGE      Возрастная доля занятых владельцами квартир, построенных до 1940 года  \n",
    "        - DIS      Взвешенные расстояния до пяти Бостонских центров занятости  \n",
    "        - RAD      Индекс доступности радиальных магистралей  \n",
    "        - TAX      НАЛОГ на недвижимость с полной стоимостью-ставка налога на 10 000 долларов  \n",
    "        - PTRATIO  Соотношение учеников и учителей по городам  \n",
    "        - B        Доля чернокожих по городам  \n",
    "        - LSTAT    % населения с низким социальным статусом  \n",
    "        - PRICE    Медианная стоимость домов, занятых владельцами, в $1000 долларах  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CRIM</th>\n",
       "      <th>ZN</th>\n",
       "      <th>INDUS</th>\n",
       "      <th>CHAS</th>\n",
       "      <th>NOX</th>\n",
       "      <th>RM</th>\n",
       "      <th>AGE</th>\n",
       "      <th>DIS</th>\n",
       "      <th>RAD</th>\n",
       "      <th>TAX</th>\n",
       "      <th>PTRATIO</th>\n",
       "      <th>B</th>\n",
       "      <th>LSTAT</th>\n",
       "      <th>PRICE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.00632</td>\n",
       "      <td>18.0</td>\n",
       "      <td>2.31</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.538</td>\n",
       "      <td>6.575</td>\n",
       "      <td>65.2</td>\n",
       "      <td>4.0900</td>\n",
       "      <td>1.0</td>\n",
       "      <td>296.0</td>\n",
       "      <td>15.3</td>\n",
       "      <td>396.90</td>\n",
       "      <td>4.98</td>\n",
       "      <td>24.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.02731</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>6.421</td>\n",
       "      <td>78.9</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2.0</td>\n",
       "      <td>242.0</td>\n",
       "      <td>17.8</td>\n",
       "      <td>396.90</td>\n",
       "      <td>9.14</td>\n",
       "      <td>21.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.02729</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>7.185</td>\n",
       "      <td>61.1</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2.0</td>\n",
       "      <td>242.0</td>\n",
       "      <td>17.8</td>\n",
       "      <td>392.83</td>\n",
       "      <td>4.03</td>\n",
       "      <td>34.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.03237</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>6.998</td>\n",
       "      <td>45.8</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3.0</td>\n",
       "      <td>222.0</td>\n",
       "      <td>18.7</td>\n",
       "      <td>394.63</td>\n",
       "      <td>2.94</td>\n",
       "      <td>33.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.06905</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>7.147</td>\n",
       "      <td>54.2</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3.0</td>\n",
       "      <td>222.0</td>\n",
       "      <td>18.7</td>\n",
       "      <td>396.90</td>\n",
       "      <td>5.33</td>\n",
       "      <td>36.2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      CRIM    ZN  INDUS  CHAS    NOX     RM   AGE     DIS  RAD    TAX  \\\n",
       "0  0.00632  18.0   2.31   0.0  0.538  6.575  65.2  4.0900  1.0  296.0   \n",
       "1  0.02731   0.0   7.07   0.0  0.469  6.421  78.9  4.9671  2.0  242.0   \n",
       "2  0.02729   0.0   7.07   0.0  0.469  7.185  61.1  4.9671  2.0  242.0   \n",
       "3  0.03237   0.0   2.18   0.0  0.458  6.998  45.8  6.0622  3.0  222.0   \n",
       "4  0.06905   0.0   2.18   0.0  0.458  7.147  54.2  6.0622  3.0  222.0   \n",
       "\n",
       "   PTRATIO       B  LSTAT  PRICE  \n",
       "0     15.3  396.90   4.98   24.0  \n",
       "1     17.8  396.90   9.14   21.6  \n",
       "2     17.8  392.83   4.03   34.7  \n",
       "3     18.7  394.63   2.94   33.4  \n",
       "4     18.7  396.90   5.33   36.2  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_boston[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Процент незаполненных данных в признаках\n",
      "CRIM: 0.0%\n",
      "ZN: 0.0%\n",
      "INDUS: 0.0%\n",
      "CHAS: 0.0%\n",
      "NOX: 0.0%\n",
      "RM: 0.0%\n",
      "AGE: 0.0%\n",
      "DIS: 0.0%\n",
      "RAD: 0.0%\n",
      "TAX: 0.0%\n",
      "PTRATIO: 0.0%\n",
      "B: 0.0%\n",
      "LSTAT: 0.0%\n",
      "PRICE: 0.0%\n"
     ]
    }
   ],
   "source": [
    "print('Процент незаполненных данных в признаках')\n",
    "for column_name in df_boston.columns:\n",
    "    col_nan_stat = round(df_boston[column_name].isna().mean() * 100, 2)\n",
    "    print(f'{column_name}: {col_nan_stat}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Посмотрим как будет работать линейная регрессия с необработанными данными"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler = StandardScaler()\n",
    "lr = LinearRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 4.6386899261728205\n",
      "R2: 0.7112260057484932\n"
     ]
    }
   ],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(df_boston[df_boston.columns.drop('PRICE')],\n",
    "                                                    df_boston['PRICE'], test_size=0.3,\n",
    "                                                    random_state=42)\n",
    "scaler.fit(X_train)\n",
    "X_train_std = scaler.transform(X_train)\n",
    "X_test_std = scaler.transform(X_test)\n",
    "lr.fit(X_train_std, y_train)\n",
    "y_predict = lr.predict(X_test_std)\n",
    "rmse = np.sqrt(mean_squared_error(y_test, y_predict))\n",
    "r2 = r2_score(y_test, y_predict)\n",
    "print('RMSE:', rmse)\n",
    "print('R2:', r2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь посмотрим таблицу кореляций признаков между собой. Выбираем коэффициент корреляции Спирмена, который показывает в том числе и нелинейную зависимость признаков."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CRIM</th>\n",
       "      <th>ZN</th>\n",
       "      <th>INDUS</th>\n",
       "      <th>CHAS</th>\n",
       "      <th>NOX</th>\n",
       "      <th>RM</th>\n",
       "      <th>AGE</th>\n",
       "      <th>DIS</th>\n",
       "      <th>RAD</th>\n",
       "      <th>TAX</th>\n",
       "      <th>PTRATIO</th>\n",
       "      <th>B</th>\n",
       "      <th>LSTAT</th>\n",
       "      <th>PRICE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>CRIM</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.571660</td>\n",
       "      <td>0.735524</td>\n",
       "      <td>0.041537</td>\n",
       "      <td>0.821465</td>\n",
       "      <td>-0.309116</td>\n",
       "      <td>0.704140</td>\n",
       "      <td>-0.744986</td>\n",
       "      <td>0.727807</td>\n",
       "      <td>0.729045</td>\n",
       "      <td>0.465283</td>\n",
       "      <td>-0.360555</td>\n",
       "      <td>0.634760</td>\n",
       "      <td>-0.558891</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ZN</th>\n",
       "      <td>-0.571660</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.642811</td>\n",
       "      <td>-0.041937</td>\n",
       "      <td>-0.634828</td>\n",
       "      <td>0.361074</td>\n",
       "      <td>-0.544423</td>\n",
       "      <td>0.614627</td>\n",
       "      <td>-0.278767</td>\n",
       "      <td>-0.371394</td>\n",
       "      <td>-0.448475</td>\n",
       "      <td>0.163135</td>\n",
       "      <td>-0.490074</td>\n",
       "      <td>0.438179</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>INDUS</th>\n",
       "      <td>0.735524</td>\n",
       "      <td>-0.642811</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.089841</td>\n",
       "      <td>0.791189</td>\n",
       "      <td>-0.415301</td>\n",
       "      <td>0.679487</td>\n",
       "      <td>-0.757080</td>\n",
       "      <td>0.455507</td>\n",
       "      <td>0.664361</td>\n",
       "      <td>0.433710</td>\n",
       "      <td>-0.285840</td>\n",
       "      <td>0.638747</td>\n",
       "      <td>-0.578255</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CHAS</th>\n",
       "      <td>0.041537</td>\n",
       "      <td>-0.041937</td>\n",
       "      <td>0.089841</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.068426</td>\n",
       "      <td>0.058813</td>\n",
       "      <td>0.067792</td>\n",
       "      <td>-0.080248</td>\n",
       "      <td>0.024579</td>\n",
       "      <td>-0.044486</td>\n",
       "      <td>-0.136065</td>\n",
       "      <td>-0.039810</td>\n",
       "      <td>-0.050575</td>\n",
       "      <td>0.140612</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NOX</th>\n",
       "      <td>0.821465</td>\n",
       "      <td>-0.634828</td>\n",
       "      <td>0.791189</td>\n",
       "      <td>0.068426</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.310344</td>\n",
       "      <td>0.795153</td>\n",
       "      <td>-0.880015</td>\n",
       "      <td>0.586429</td>\n",
       "      <td>0.649527</td>\n",
       "      <td>0.391309</td>\n",
       "      <td>-0.296662</td>\n",
       "      <td>0.636828</td>\n",
       "      <td>-0.562609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RM</th>\n",
       "      <td>-0.309116</td>\n",
       "      <td>0.361074</td>\n",
       "      <td>-0.415301</td>\n",
       "      <td>0.058813</td>\n",
       "      <td>-0.310344</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.278082</td>\n",
       "      <td>0.263168</td>\n",
       "      <td>-0.107492</td>\n",
       "      <td>-0.271898</td>\n",
       "      <td>-0.312923</td>\n",
       "      <td>0.053660</td>\n",
       "      <td>-0.640832</td>\n",
       "      <td>0.633576</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AGE</th>\n",
       "      <td>0.704140</td>\n",
       "      <td>-0.544423</td>\n",
       "      <td>0.679487</td>\n",
       "      <td>0.067792</td>\n",
       "      <td>0.795153</td>\n",
       "      <td>-0.278082</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.801610</td>\n",
       "      <td>0.417983</td>\n",
       "      <td>0.526366</td>\n",
       "      <td>0.355384</td>\n",
       "      <td>-0.228022</td>\n",
       "      <td>0.657071</td>\n",
       "      <td>-0.547562</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DIS</th>\n",
       "      <td>-0.744986</td>\n",
       "      <td>0.614627</td>\n",
       "      <td>-0.757080</td>\n",
       "      <td>-0.080248</td>\n",
       "      <td>-0.880015</td>\n",
       "      <td>0.263168</td>\n",
       "      <td>-0.801610</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.495806</td>\n",
       "      <td>-0.574336</td>\n",
       "      <td>-0.322041</td>\n",
       "      <td>0.249595</td>\n",
       "      <td>-0.564262</td>\n",
       "      <td>0.445857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RAD</th>\n",
       "      <td>0.727807</td>\n",
       "      <td>-0.278767</td>\n",
       "      <td>0.455507</td>\n",
       "      <td>0.024579</td>\n",
       "      <td>0.586429</td>\n",
       "      <td>-0.107492</td>\n",
       "      <td>0.417983</td>\n",
       "      <td>-0.495806</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.704876</td>\n",
       "      <td>0.318330</td>\n",
       "      <td>-0.282533</td>\n",
       "      <td>0.394322</td>\n",
       "      <td>-0.346776</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TAX</th>\n",
       "      <td>0.729045</td>\n",
       "      <td>-0.371394</td>\n",
       "      <td>0.664361</td>\n",
       "      <td>-0.044486</td>\n",
       "      <td>0.649527</td>\n",
       "      <td>-0.271898</td>\n",
       "      <td>0.526366</td>\n",
       "      <td>-0.574336</td>\n",
       "      <td>0.704876</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.453345</td>\n",
       "      <td>-0.329843</td>\n",
       "      <td>0.534423</td>\n",
       "      <td>-0.562411</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PTRATIO</th>\n",
       "      <td>0.465283</td>\n",
       "      <td>-0.448475</td>\n",
       "      <td>0.433710</td>\n",
       "      <td>-0.136065</td>\n",
       "      <td>0.391309</td>\n",
       "      <td>-0.312923</td>\n",
       "      <td>0.355384</td>\n",
       "      <td>-0.322041</td>\n",
       "      <td>0.318330</td>\n",
       "      <td>0.453345</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.072027</td>\n",
       "      <td>0.467259</td>\n",
       "      <td>-0.555905</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>B</th>\n",
       "      <td>-0.360555</td>\n",
       "      <td>0.163135</td>\n",
       "      <td>-0.285840</td>\n",
       "      <td>-0.039810</td>\n",
       "      <td>-0.296662</td>\n",
       "      <td>0.053660</td>\n",
       "      <td>-0.228022</td>\n",
       "      <td>0.249595</td>\n",
       "      <td>-0.282533</td>\n",
       "      <td>-0.329843</td>\n",
       "      <td>-0.072027</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.210562</td>\n",
       "      <td>0.185664</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LSTAT</th>\n",
       "      <td>0.634760</td>\n",
       "      <td>-0.490074</td>\n",
       "      <td>0.638747</td>\n",
       "      <td>-0.050575</td>\n",
       "      <td>0.636828</td>\n",
       "      <td>-0.640832</td>\n",
       "      <td>0.657071</td>\n",
       "      <td>-0.564262</td>\n",
       "      <td>0.394322</td>\n",
       "      <td>0.534423</td>\n",
       "      <td>0.467259</td>\n",
       "      <td>-0.210562</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.852914</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PRICE</th>\n",
       "      <td>-0.558891</td>\n",
       "      <td>0.438179</td>\n",
       "      <td>-0.578255</td>\n",
       "      <td>0.140612</td>\n",
       "      <td>-0.562609</td>\n",
       "      <td>0.633576</td>\n",
       "      <td>-0.547562</td>\n",
       "      <td>0.445857</td>\n",
       "      <td>-0.346776</td>\n",
       "      <td>-0.562411</td>\n",
       "      <td>-0.555905</td>\n",
       "      <td>0.185664</td>\n",
       "      <td>-0.852914</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             CRIM        ZN     INDUS      CHAS       NOX        RM       AGE  \\\n",
       "CRIM     1.000000 -0.571660  0.735524  0.041537  0.821465 -0.309116  0.704140   \n",
       "ZN      -0.571660  1.000000 -0.642811 -0.041937 -0.634828  0.361074 -0.544423   \n",
       "INDUS    0.735524 -0.642811  1.000000  0.089841  0.791189 -0.415301  0.679487   \n",
       "CHAS     0.041537 -0.041937  0.089841  1.000000  0.068426  0.058813  0.067792   \n",
       "NOX      0.821465 -0.634828  0.791189  0.068426  1.000000 -0.310344  0.795153   \n",
       "RM      -0.309116  0.361074 -0.415301  0.058813 -0.310344  1.000000 -0.278082   \n",
       "AGE      0.704140 -0.544423  0.679487  0.067792  0.795153 -0.278082  1.000000   \n",
       "DIS     -0.744986  0.614627 -0.757080 -0.080248 -0.880015  0.263168 -0.801610   \n",
       "RAD      0.727807 -0.278767  0.455507  0.024579  0.586429 -0.107492  0.417983   \n",
       "TAX      0.729045 -0.371394  0.664361 -0.044486  0.649527 -0.271898  0.526366   \n",
       "PTRATIO  0.465283 -0.448475  0.433710 -0.136065  0.391309 -0.312923  0.355384   \n",
       "B       -0.360555  0.163135 -0.285840 -0.039810 -0.296662  0.053660 -0.228022   \n",
       "LSTAT    0.634760 -0.490074  0.638747 -0.050575  0.636828 -0.640832  0.657071   \n",
       "PRICE   -0.558891  0.438179 -0.578255  0.140612 -0.562609  0.633576 -0.547562   \n",
       "\n",
       "              DIS       RAD       TAX   PTRATIO         B     LSTAT     PRICE  \n",
       "CRIM    -0.744986  0.727807  0.729045  0.465283 -0.360555  0.634760 -0.558891  \n",
       "ZN       0.614627 -0.278767 -0.371394 -0.448475  0.163135 -0.490074  0.438179  \n",
       "INDUS   -0.757080  0.455507  0.664361  0.433710 -0.285840  0.638747 -0.578255  \n",
       "CHAS    -0.080248  0.024579 -0.044486 -0.136065 -0.039810 -0.050575  0.140612  \n",
       "NOX     -0.880015  0.586429  0.649527  0.391309 -0.296662  0.636828 -0.562609  \n",
       "RM       0.263168 -0.107492 -0.271898 -0.312923  0.053660 -0.640832  0.633576  \n",
       "AGE     -0.801610  0.417983  0.526366  0.355384 -0.228022  0.657071 -0.547562  \n",
       "DIS      1.000000 -0.495806 -0.574336 -0.322041  0.249595 -0.564262  0.445857  \n",
       "RAD     -0.495806  1.000000  0.704876  0.318330 -0.282533  0.394322 -0.346776  \n",
       "TAX     -0.574336  0.704876  1.000000  0.453345 -0.329843  0.534423 -0.562411  \n",
       "PTRATIO -0.322041  0.318330  0.453345  1.000000 -0.072027  0.467259 -0.555905  \n",
       "B        0.249595 -0.282533 -0.329843 -0.072027  1.000000 -0.210562  0.185664  \n",
       "LSTAT   -0.564262  0.394322  0.534423  0.467259 -0.210562  1.000000 -0.852914  \n",
       "PRICE    0.445857 -0.346776 -0.562411 -0.555905  0.185664 -0.852914  1.000000  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_boston.corr(method='spearman')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "По таблице корреляции делаем следующие выводы\n",
    "\n",
    "1. Наибольшее влияние на целевую переменную PRICE оказывают признаки LSTAT и RM, поэтому их можно назвать ключевыми признаками.\n",
    "2. Признак CRIM имеет достаточную корреляцию с признаками INDUS (0.74), NOX (0.82), AGE (0.70), DIS(0.74), RAD (0.72), TAX (0.73), имеет смысл удалить эти признаки.\n",
    "\n",
    "Пробуем удалить признаки с достаточно высокой корреляцией и посмотрим как работает линейная модель регрессии."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CRIM</th>\n",
       "      <th>ZN</th>\n",
       "      <th>CHAS</th>\n",
       "      <th>RM</th>\n",
       "      <th>PTRATIO</th>\n",
       "      <th>B</th>\n",
       "      <th>LSTAT</th>\n",
       "      <th>PRICE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.00632</td>\n",
       "      <td>18.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.575</td>\n",
       "      <td>15.3</td>\n",
       "      <td>396.90</td>\n",
       "      <td>4.98</td>\n",
       "      <td>24.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.02731</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.421</td>\n",
       "      <td>17.8</td>\n",
       "      <td>396.90</td>\n",
       "      <td>9.14</td>\n",
       "      <td>21.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.02729</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.185</td>\n",
       "      <td>17.8</td>\n",
       "      <td>392.83</td>\n",
       "      <td>4.03</td>\n",
       "      <td>34.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.03237</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.998</td>\n",
       "      <td>18.7</td>\n",
       "      <td>394.63</td>\n",
       "      <td>2.94</td>\n",
       "      <td>33.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.06905</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.147</td>\n",
       "      <td>18.7</td>\n",
       "      <td>396.90</td>\n",
       "      <td>5.33</td>\n",
       "      <td>36.2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      CRIM    ZN  CHAS     RM  PTRATIO       B  LSTAT  PRICE\n",
       "0  0.00632  18.0   0.0  6.575     15.3  396.90   4.98   24.0\n",
       "1  0.02731   0.0   0.0  6.421     17.8  396.90   9.14   21.6\n",
       "2  0.02729   0.0   0.0  7.185     17.8  392.83   4.03   34.7\n",
       "3  0.03237   0.0   0.0  6.998     18.7  394.63   2.94   33.4\n",
       "4  0.06905   0.0   0.0  7.147     18.7  396.90   5.33   36.2"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_boston_no_corr = df_boston[df_boston.columns.drop(['INDUS', 'NOX', 'AGE', 'DIS', 'RAD', 'TAX'])]\n",
    "df_boston_no_corr[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 5.163634482459411\n",
      "R2: 0.6421686568616799\n"
     ]
    }
   ],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(df_boston_no_corr[df_boston_no_corr.columns.drop('PRICE')],\n",
    "                                                    df_boston_no_corr['PRICE'], test_size=0.3,\n",
    "                                                    random_state=42)\n",
    "scaler.fit(X_train)\n",
    "X_train_std = scaler.transform(X_train)\n",
    "X_test_std = scaler.transform(X_test)\n",
    "lr.fit(X_train_std, y_train)\n",
    "y_predict = lr.predict(X_test_std)\n",
    "rmse = np.sqrt(mean_squared_error(y_test, y_predict))\n",
    "r2 = r2_score(y_test, y_predict)\n",
    "print('RMSE:', rmse)\n",
    "print('R2:', r2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "По сравнению с необработанной моделью ошибка увеличилась.\n",
    "\n",
    "Теперь посмотрим что произойдет, если оставить только значимые с чочки зрения корреляции признаки LSTAT и RM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 5.460428346919539\n",
      "R2: 0.59985184477156\n"
     ]
    }
   ],
   "source": [
    "df_boston_main = df_boston[['LSTAT', 'RM', 'PRICE']]\n",
    "X_train, X_test, y_train, y_test = train_test_split(df_boston_main[df_boston_main.columns.drop('PRICE')],\n",
    "                                                    df_boston_main['PRICE'], test_size=0.3,\n",
    "                                                    random_state=42)\n",
    "scaler.fit(X_train)\n",
    "X_train_std = scaler.transform(X_train)\n",
    "X_test_std = scaler.transform(X_test)\n",
    "lr.fit(X_train_std, y_train)\n",
    "y_predict = lr.predict(X_test_std)\n",
    "rmse = np.sqrt(mean_squared_error(y_test, y_predict))\n",
    "r2 = r2_score(y_test, y_predict)\n",
    "print('RMSE:', rmse)\n",
    "print('R2:', r2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Получили еще более худший результат, поэтому принимаем решение оставлять все признаки в датасете.\n",
    "\n",
    "Теперь проанализируем на качество данных каждый признак в датасете."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CRIM</th>\n",
       "      <th>ZN</th>\n",
       "      <th>INDUS</th>\n",
       "      <th>CHAS</th>\n",
       "      <th>NOX</th>\n",
       "      <th>RM</th>\n",
       "      <th>AGE</th>\n",
       "      <th>DIS</th>\n",
       "      <th>RAD</th>\n",
       "      <th>TAX</th>\n",
       "      <th>PTRATIO</th>\n",
       "      <th>B</th>\n",
       "      <th>LSTAT</th>\n",
       "      <th>PRICE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>506.000000</td>\n",
       "      <td>506.000000</td>\n",
       "      <td>506.000000</td>\n",
       "      <td>506.000000</td>\n",
       "      <td>506.000000</td>\n",
       "      <td>506.000000</td>\n",
       "      <td>506.000000</td>\n",
       "      <td>506.000000</td>\n",
       "      <td>506.000000</td>\n",
       "      <td>506.000000</td>\n",
       "      <td>506.000000</td>\n",
       "      <td>506.000000</td>\n",
       "      <td>506.000000</td>\n",
       "      <td>506.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>3.613524</td>\n",
       "      <td>11.363636</td>\n",
       "      <td>11.136779</td>\n",
       "      <td>0.069170</td>\n",
       "      <td>0.554695</td>\n",
       "      <td>6.284634</td>\n",
       "      <td>68.574901</td>\n",
       "      <td>3.795043</td>\n",
       "      <td>9.549407</td>\n",
       "      <td>408.237154</td>\n",
       "      <td>18.455534</td>\n",
       "      <td>356.674032</td>\n",
       "      <td>12.653063</td>\n",
       "      <td>22.532806</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>8.601545</td>\n",
       "      <td>23.322453</td>\n",
       "      <td>6.860353</td>\n",
       "      <td>0.253994</td>\n",
       "      <td>0.115878</td>\n",
       "      <td>0.702617</td>\n",
       "      <td>28.148861</td>\n",
       "      <td>2.105710</td>\n",
       "      <td>8.707259</td>\n",
       "      <td>168.537116</td>\n",
       "      <td>2.164946</td>\n",
       "      <td>91.294864</td>\n",
       "      <td>7.141062</td>\n",
       "      <td>9.197104</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.006320</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.460000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.385000</td>\n",
       "      <td>3.561000</td>\n",
       "      <td>2.900000</td>\n",
       "      <td>1.129600</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>187.000000</td>\n",
       "      <td>12.600000</td>\n",
       "      <td>0.320000</td>\n",
       "      <td>1.730000</td>\n",
       "      <td>5.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.082045</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5.190000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.449000</td>\n",
       "      <td>5.885500</td>\n",
       "      <td>45.025000</td>\n",
       "      <td>2.100175</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>279.000000</td>\n",
       "      <td>17.400000</td>\n",
       "      <td>375.377500</td>\n",
       "      <td>6.950000</td>\n",
       "      <td>17.025000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.256510</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>9.690000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.538000</td>\n",
       "      <td>6.208500</td>\n",
       "      <td>77.500000</td>\n",
       "      <td>3.207450</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>330.000000</td>\n",
       "      <td>19.050000</td>\n",
       "      <td>391.440000</td>\n",
       "      <td>11.360000</td>\n",
       "      <td>21.200000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>3.677083</td>\n",
       "      <td>12.500000</td>\n",
       "      <td>18.100000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.624000</td>\n",
       "      <td>6.623500</td>\n",
       "      <td>94.075000</td>\n",
       "      <td>5.188425</td>\n",
       "      <td>24.000000</td>\n",
       "      <td>666.000000</td>\n",
       "      <td>20.200000</td>\n",
       "      <td>396.225000</td>\n",
       "      <td>16.955000</td>\n",
       "      <td>25.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>88.976200</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>27.740000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.871000</td>\n",
       "      <td>8.780000</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>12.126500</td>\n",
       "      <td>24.000000</td>\n",
       "      <td>711.000000</td>\n",
       "      <td>22.000000</td>\n",
       "      <td>396.900000</td>\n",
       "      <td>37.970000</td>\n",
       "      <td>50.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             CRIM          ZN       INDUS        CHAS         NOX          RM  \\\n",
       "count  506.000000  506.000000  506.000000  506.000000  506.000000  506.000000   \n",
       "mean     3.613524   11.363636   11.136779    0.069170    0.554695    6.284634   \n",
       "std      8.601545   23.322453    6.860353    0.253994    0.115878    0.702617   \n",
       "min      0.006320    0.000000    0.460000    0.000000    0.385000    3.561000   \n",
       "25%      0.082045    0.000000    5.190000    0.000000    0.449000    5.885500   \n",
       "50%      0.256510    0.000000    9.690000    0.000000    0.538000    6.208500   \n",
       "75%      3.677083   12.500000   18.100000    0.000000    0.624000    6.623500   \n",
       "max     88.976200  100.000000   27.740000    1.000000    0.871000    8.780000   \n",
       "\n",
       "              AGE         DIS         RAD         TAX     PTRATIO           B  \\\n",
       "count  506.000000  506.000000  506.000000  506.000000  506.000000  506.000000   \n",
       "mean    68.574901    3.795043    9.549407  408.237154   18.455534  356.674032   \n",
       "std     28.148861    2.105710    8.707259  168.537116    2.164946   91.294864   \n",
       "min      2.900000    1.129600    1.000000  187.000000   12.600000    0.320000   \n",
       "25%     45.025000    2.100175    4.000000  279.000000   17.400000  375.377500   \n",
       "50%     77.500000    3.207450    5.000000  330.000000   19.050000  391.440000   \n",
       "75%     94.075000    5.188425   24.000000  666.000000   20.200000  396.225000   \n",
       "max    100.000000   12.126500   24.000000  711.000000   22.000000  396.900000   \n",
       "\n",
       "            LSTAT       PRICE  \n",
       "count  506.000000  506.000000  \n",
       "mean    12.653063   22.532806  \n",
       "std      7.141062    9.197104  \n",
       "min      1.730000    5.000000  \n",
       "25%      6.950000   17.025000  \n",
       "50%     11.360000   21.200000  \n",
       "75%     16.955000   25.000000  \n",
       "max     37.970000   50.000000  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_boston.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Признак CRIM имеет значительные выбросы, максимальное значение 88.976200 при значении в третьем квартиле 3.677083."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Количество образцов с CRIM > 9: 66\n"
     ]
    }
   ],
   "source": [
    "print('Количество образцов с CRIM > 9:', len(df_boston[df_boston['CRIM'] > 9]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Так как количество строк с CRIM > 9 всего 66, а замена на медианные значения не улучшают нашу модель, удалим эти данные из датасета и проверим качество модели."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_boston = df_boston[df_boston['CRIM'] <= 9]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 3.802132689962362\n",
      "R2: 0.7219662854637542\n"
     ]
    }
   ],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(df_boston[df_boston.columns.drop('PRICE')],\n",
    "                                                    df_boston['PRICE'], test_size=0.3,\n",
    "                                                    random_state=42)\n",
    "scaler.fit(X_train)\n",
    "X_train_std = scaler.transform(X_train)\n",
    "X_test_std = scaler.transform(X_test)\n",
    "lr.fit(X_train_std, y_train)\n",
    "y_predict = lr.predict(X_test_std)\n",
    "rmse = np.sqrt(mean_squared_error(y_test, y_predict))\n",
    "r2 = r2_score(y_test, y_predict)\n",
    "print('RMSE:', rmse)\n",
    "print('R2:', r2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Мы получили улучшение работы модели с 4.639 до 3.802.\n",
    "\n",
    "Признак ZN так же имеет аномальные выбросы с максимальным значением 100 при значении 3 квартиля в 12.5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Количество образцов с ZN > 20: 101\n"
     ]
    }
   ],
   "source": [
    "print('Количество образцов с ZN > 20:', len(df_boston[df_boston.ZN > 20]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Удаление образцов с ZN > 20 и заполнение их медианой приводит к ухудшению результата. Поэтому оставляем признак как есть.\n",
    "\n",
    "У признака AGE так же есть выбросы в сторону минимального значения. первый квартиль имеет значение 45. А минимальное значение 2.9."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Количество образцов с AGE < 20: 34\n"
     ]
    }
   ],
   "source": [
    "print('Количество образцов с AGE < 20:', len(df_boston[df_boston.AGE < 20]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Удаление этих данных ухудшает нашу модель по всем параметрам. Пробуем заполнить значения AGE < 30 медианным значением."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_boston[df_boston.AGE < 30] = df_boston.AGE.median()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 3.711986723724966\n",
      "R2: 0.9642603407813976\n"
     ]
    }
   ],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(df_boston[df_boston.columns.drop('PRICE')],\n",
    "                                                    df_boston['PRICE'], test_size=0.3,\n",
    "                                                    random_state=42)\n",
    "scaler.fit(X_train)\n",
    "X_train_std = scaler.transform(X_train)\n",
    "X_test_std = scaler.transform(X_test)\n",
    "lr.fit(X_train_std, y_train)\n",
    "y_predict = lr.predict(X_test_std)\n",
    "rmse = np.sqrt(mean_squared_error(y_test, y_predict))\n",
    "r2 = r2_score(y_test, y_predict)\n",
    "print('RMSE:', rmse)\n",
    "print('R2:', r2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Значение RMSE немного улучшилось с 3.802 до 3.711, а так же значительно улучшилось качество модели по сравнению с моделью усреднения с 0.721 до 0.964.\n",
    "\n",
    "Теперь попробуем построить модель с помощью полиномиальной регрессии. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 3.2839318684694314\n",
      "R2: 0.9720278495691006\n"
     ]
    }
   ],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(df_boston[df_boston.columns.drop('PRICE')],\n",
    "                                                    df_boston['PRICE'], test_size=0.3,\n",
    "                                                    random_state=42)\n",
    "\n",
    "polynom = PolynomialFeatures(2)\n",
    "polynom.fit(X_train)\n",
    "X_train_poly = polynom.transform(X_train)\n",
    "X_test_poly = polynom.transform(X_test)\n",
    "\n",
    "scaler.fit(X_train_poly)\n",
    "X_train_poly_std = scaler.transform(X_train_poly)\n",
    "X_test_poly_std = scaler.transform(X_test_poly)\n",
    "\n",
    "lr.fit(X_train_poly_std, y_train)\n",
    "y_predict = lr.predict(X_test_poly_std)\n",
    "\n",
    "rmse = np.sqrt(mean_squared_error(y_test, y_predict))\n",
    "r2 = r2_score(y_test, y_predict)\n",
    "print('RMSE:', rmse)\n",
    "print('R2:', r2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "С помощью полинома нам удалось улучшить RMSE с 3.712 до 3.284, а R2 с 0.964 до 0.972\n",
    "Найдем признаки, которые оказывают наибольшее влияние на модель полиномиальной регрессии."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(69501.59813184377, 'x3 x4'),\n",
       " (68609.91074812133, 'x3^2'),\n",
       " (53642.05308861647, 'x4^2'),\n",
       " (28137.31751480733, 'x4 x5'),\n",
       " (22712.089876973856, 'x4 x7'),\n",
       " (12798.514508258979, 'x4 x10'),\n",
       " (7173.877170452307, 'x0 x3'),\n",
       " (5095.775604926884, 'x3 x5'),\n",
       " (4959.788391879584, 'x0 x4'),\n",
       " (4772.2809057023, 'x3 x7')]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "poly_f_names = polynom.get_feature_names()\n",
    "poly_f_coef = lr.coef_\n",
    "poly_f_sorted_index = np.argsort(poly_f_coef)[::-1]\n",
    "\n",
    "sorted([(np.abs(coef), name) for coef, name in zip (lr.coef_, polynom.get_feature_names())], \n",
    "       reverse=True)[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь попробуем улучшить наш датасет с помощью новых признаков из полиномиальной регрессии."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_boston['CHAS*NOX'] = df_boston['CHAS'] * df_boston['NOX']\n",
    "df_boston['CHAS^2'] = df_boston['CHAS'] ** 2\n",
    "df_boston['NOX^2'] = df_boston['NOX'] ** 2\n",
    "df_boston['NOX*RM'] = df_boston['NOX'] * df_boston['RM']\n",
    "df_boston['NOX*DIS'] = df_boston['NOX'] * df_boston['DIS']\n",
    "df_boston['NOX*PTRATIO'] = df_boston['NOX'] * df_boston['PTRATIO']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 3.464966715232604\n",
      "R2: 0.9688587735331491\n"
     ]
    }
   ],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(df_boston[df_boston.columns.drop('PRICE')],\n",
    "                                                    df_boston['PRICE'], test_size=0.3,\n",
    "                                                    random_state=42)\n",
    "scaler.fit(X_train)\n",
    "X_train_std = scaler.transform(X_train)\n",
    "X_test_std = scaler.transform(X_test)\n",
    "lr.fit(X_train_std, y_train)\n",
    "y_predict = lr.predict(X_test_std)\n",
    "rmse = np.sqrt(mean_squared_error(y_test, y_predict))\n",
    "r2 = r2_score(y_test, y_predict)\n",
    "print('RMSE:', rmse)\n",
    "print('R2:', r2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "До уровня полиномиальной регресиси добраться не удалось, но тем не менее мы RMSE с 3.712 до 3.465, а R2 с 0.964 до 0.969.\n",
    "\n",
    "В итоге нам удалось улучшить результат с\n",
    "\n",
    "    RMSE: 4.6386899261728205\n",
    "    R2: 0.7112260057484932\n",
    "до   \n",
    "\n",
    "    RMSE: 3.464966715232604\n",
    "    R2: 0.9688587735331491\n",
    "    \n",
    "с помощью обработки признаков и добавления к ним новых значимых признаков из полиномиальной регрессии и до \n",
    "\n",
    "    RMSE: 3.2839318684694314\n",
    "    R2: 0.9720278495691006\n",
    "    \n",
    " с помощью полиномиальной регрессии"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
